The invention relates to an image processing method and apparatus and, more particularly, to an image processing method and apparatus dealing with a digital image encoded by block.
Conventionally, in an image processing system which stores encoded image data and displays it on a monitor or the like, in order to obtain image data representing an image whose size is different from an original image, it has been common to perform enlargement or reduction after decoding the compressed original image data.
For example, to obtain an image of a smaller size (a reduced image) than the original image, the coded original image data is reproduced in an original size, then known interpolation processing, such as sub-sampling, has to be performed to complete the enlargement or reduction.
Conventional decoding and displaying processes will be explained below with reference to drawings.
FIG. 33 is a block diagram illustrating a constitution of a decoding and displaying processes in the conventional image processing apparatus.
In FIG. 33, encoded data, based on a discrete cosine transfer (DCT) using a block size of eight-pixel by eight-pixel, which is a block size of a still image standard encoding method, is stored in an image memory A 1501. The coded data is inputted to a decoding unit 1502, thereat the coded data is decoded in accordance with a decoding processing which will be described later. Then, the decoded data is stored in an image memory B 1503. Next, the decoded image data stored in the image memory B 1503 is inputted to an enlargement/reduction processing unit 1504 where the image data is applied to an enlargement/reduction process, such as a sub-sampling or such interpolation processing, based on an enlargement/reduction ratio designated by an enlargement/reduction ratio designator 1507. After the processes, the image data is stored in an image memory C 1505 for display (a display memory). Then, the processed data stored in the image memory C 1505 is displayed on a display 1506.
In FIG. 34, an example of a conventional decoding unit 1502 shown in FIG. 33 is illustrated.
In the decoding unit 1502, an 8.times.8 DCT coded data is decoded. In FIG. 34, the coded data is inputted to a Huffman decoding unit 1601 via an input terminal 1600, and decoded, then 8.times.8 DCT coefficients are reproduced. Next, the reproduced DCT coefficients are inputted into an inverse DCT unit 1602 and applied with an inverse DCT process, then an image data of an 8.times.8 block size is reproduced. The reproduced image data is outputted to the image memory B 1503 in FIG. 33 via an output terminal 1603.
As described above, in a conventional image processing method and apparatus, all the DCT coded data is converted to image data of an original size by decoding the coded data, then enlarged or reduced.
However, according to the conventional method and apparatus described above, it takes considerable time to decode the encoded original image data. Further, it requires a memory of large capacity, such as the image memory B 1503 shown in FIG. 33, for temporary storing the original image data while processing it.
Therefore, by utilizing a hierarchical coding method having plural kinds of image data of different sizes in a single coded data, it is possible to encode at high speed without a memory of large capacity for temporary storing data.
As a typical hierarchical coding method used by a conventional image processing apparatus, there is the well-known Laplacean Pyramid coding method.
According to the Laplacean Pyramid coding method, an input image is hierarchically compressed from 1/2, 1/4, 1/8, and so on, and at each hierarchy of the pyramid, a compressed image is coded by using the discrete cosine transfer, or the like. The order of encoding is to encode the most compressed image, then encode the difference between the image expanded to doubled the size of the decoded image corresponding to the most compressed image and the image compressed to the aforesaid "doubled" size obtained from the input image. The aforesaid process is repeated until the image size of the hierarchy which is the subject to encode and the input image becomes the same size, thereby coded data of multi-hierarchical images can be generated.
Coded data of each hierarchy of the hierarchical coding method, such as the Laplacean pyramid coding method as described above, is generated by coding a part of the decoded original image data. Then, by decoding coded data of each hierarchy, the lower hierarchical image can be obtained as, for instance, a reduced image.
Further, not only the aforesaid hierarchical coding method but also a sequential coding method in which an input image is sequentially coded from left to right and from top to bottom is widely known. Data coded by the sequential coding method is the same as data which is coded by the hierarchical coding method using only a single hierarchy.
Referring to FIGS. 35 and 36, a conventional hierarchical coding method will be described. FIG. 35 is a block diagram illustrating a structure of an image processing apparatus capable of using the conventional hierarchical coding method.
In FIG. 35, reference numeral 101 denotes a decoder, which reproduce RGB image data from image data, encoded by a DCT sequential encoder (not shown), as an input, by performing decoding, inverse-quantizing, inverse DCT, and color conversion. The RGB image data outputted from the decoder 101 is sub-sampled to a size of 1/2 in the vertical and horizontal directions by a sub-sampling unit A 102, thereby an image of 1/4 (1/2.times.1/2) size of the original image is produced. At a sub-sampling unit B 103, the sub-sampled RGB image data is further sub-sampled to a 1/2 size in the vertical and horizontal directions, and a 1/16 (1/4.times.1/4) size image of the original image is produced. Furthermore, at a sub-sampling unit C 104, the RGB image data from the sub-sampling unit B 103 is further sub-sampled to a 1/2 size in the vertical and horizontal directions, then an image of 1/64 (1/8.times.1/8) size of the original image is obtained. The 1/8.times.1/8 size image outputted from the sub-sampling unit C 104 is encoded at a CODEC A 105 which generates coded data of 1/8.times.1/8 size. Further, the CODEC A 105 output locally decoded image data to an interpolation processing unit A 106. The interpolation processing unit A 106 applies interpolation processing on the decoded 1/8.times.1/8 size image data to expand it to a doubled size image data, then output the obtained data to a subtractor 107. The subtractor 107 finds the difference between the doubled 1/8.times.1/8 size image data outputted from the interpolation processing unit A 106 and the 1/4.times.1/4 size image data outputted from the sub-sampling unit B 103, then the obtained difference is outputted to a CODEC B 108. The CODEC B 108 encodes the 1/4.times.1/4 size image which is inputted from the subtractor 107, thus the difference between the 1/8.times.1/8 size image and the 1/4.times.1/4 size image is encoded, and 1/4.times.1/4 size coded data is generated. Further, the CODEC B 107 outputs the locally decoded image data to an adder 109. The adder 109 adds the 1/4.times.1/4 size image data from the CODEC B 108 to the 1/4.times.1/4 size image data outputted from the interpolation processing unit A 106, and the obtained sum is expanded to a doubled size image data at a interpolation processing unit B 110, then outputted to a subtractor 111. The subtractor 111 calculates difference between the doubled 1/4.times.1/4 size image data outputted from the interpolation processing unit B 110 and the 1/2.times.1/2 size image data outputted from the sub-sampling unit A 102, then the obtained difference is outputted to a CODEC C 112. The CODEC C 112 encodes the 1/2.times.1/2 size image inputted from the subtractor 111, in other words, the difference between the 1/4.times.1/4 size image and the 1/2.times.1/2 size image is encoded, then the CODEC C 112 generates 1/2.times.1/2 size coded data. Further, the CODEC C 112 outputs the locally decoded image data to an adder 113. The adder 113 adds the 1/2.times.1/2 size image data outputted from the CODEC C 112 to the 1/2.times.1/2 size image data from the interpolation processing unit B 110, the sum is expanded to a double size image data at a interpolation processing unit C 114, then outputted to a subtractor 115. The subtractor 115 finds the difference between the doubled 1/2.times.1/2 size image data outputted from the interpolation processing unit C 114 and the full size image data from the decoder 101, then the obtained difference is outputted to a CODEC D 116. The CODEC D 116 encodes the full size image inputted from the subtractor 115, in other words, the difference between the 1/2.times.1/2 size image and the full size image is encoded to generate full size coded data. The detailed configurations of CODEC A 105, the CODEC B 108, the CODEC C 112, and the CODEC D 116 in FIG. 35 are basically same except for the amount of memory which is used in each processing stage.
Next, referring to FIG. 36, the detailed configuration of a CODEC will be described. FIG. 36 is a block diagram showing the detailed configuration of the CODEC. In FIG. 36, an input image is a decoded RGB signal, and converted from the RGB signal to a luminance/color difference (YUV) signal at a color converter A 117, then applied with 8.times.8 DCT process on each color signal at a DCT unit 118. Next at a quantizing unit 119, the DCT coefficients obtained at the DCT unit 118 are linearly quantized, and are coded in accordance with the Huffman coding method at the encoding unit 120, thereby coded data is outputted. Whereas, the output from the quantizing unit 119 is also sent to an inverse-quantizing unit 121 to be inverse-quantized, and the processed data is changed to DCT coefficient data. Next at an inverse DCT unit 122, 8.times.8 inverse DCT process is applied to the DCT coefficient data, then a YUV signal is generated. Finally, the YUV signal is converted into a RGB signal at color converting unit B 123. The output via the inverse-quantizing unit 121, the inverse DCT unit 122, and the color converter B 123 are local decode output from each CODEC shown in FIG. 35.
According to the conventional hierarchical coding method as described above, sequential coding data is converted into, for instance, hierarchical coded data of four hierarchies. On the other hand, a process for converting from the hierarchical coded data into sequential coded data is performed by reversing the process which is described above.
In order to change sizes of an image represented by coded data, there is a method to obtain a n/8.times.n/8 (n.ltoreq.8; n is a positive integer) size reduced image by extracted a n.times.n matrix portion from the 8.times.8 DCT coefficients. In other words, it is possible to obtain a 1/4.times.1/4 size reduced image data by extracting 2.times.2 DCT coefficients out of 8.times.8 DCT coefficients, and a 1/2.times.1/2 size reduced image data by extracting 4.times.4 DCT coefficients out of the 8.times.8 DCT coefficients, for example. Accordingly, it is possible to consider the zigzag scanned 8.times.8 DCT coefficients as a single sequential coded data.
According to this method, the image memory B 1503 and the enlargement/reduction processing unit 1504 of the conventional configuration shown in FIG. 33 become unnecessary, and image data outputted from the inverse DCT unit 1602 is directly outputted to the image memory C 1505, shown in FIG. 33, via the output terminal 1603. Note that, in this case, the enlargement/reduction ratio designator 1507 is connected to the inverse DCT unit 1602, shown in FIG. 34, of the decoding unit 1502.
In a reduction process as this, by terminating decoding process of the remained DCT coefficients at the completion of the Huffman decoding processing of n.times.n DCT coefficients which are required for inverse DCT process when decoding the 8.times.8 DCT coefficient block, the top position of the next DCT coefficient block is lost, which results in ruining the decoding process. Therefore, the entirely encoded 8.times.8 DCT coefficients have to be decoded. Thus, the decoding process requires quite a long time, and it has not been easy to speed up the decoding process.
Furthermore, in a case where the sequential coded data is converted into hierarchical coded data as described above with reference to FIGS. 35 and 36, or in a case where the hierarchical coded data is converted into sequential coded data, a memory of large capacity is required since all the coded data has to be decoded and convert into RGB image data. In addition, there is a problem in that image quality is often deteriorated because of quantized errors caused by coding and decoding in multi-hierarchy, and errors caused by color conversion.